Approximate Farkas lemmas and stopping rules for iterative infeasible-point algorithms for linear programming

In exact arithmetic, the simplex method applied to a particular linear programming problem instance either shows that it is infeasible, shows that its dual is infeasible, or generates optimal solutions to both problems. Interiorpoint methods do not provide such clear-cut information. We provide general tools (extensions of the Farkas Lemma) for concluding that a problem or its dual is likely (in a certain well-de ned sense) to be infeasible, and apply them to develop stopping rules for a generic infeasible-interior-point method and for the homogeneous self-dual algorithm for linear programming. These rules allow precise conclusions to be drawn about the linear programming problem and its dual: either near-optimal solutions are produced, or we obtain \certi cates" that all optimal solutions, or all feasible solutions to the primal or dual, must have large norm. Our rules thus allow more de nitive interpretation of the output of such an algorithm than previous termination criteria. We give bounds on the number of iterations required before these rules apply. Our tools may also be useful for other iterative methods for linear programming. Key words: linear programming, Farkas lemma, infeasible-interior-point methods, stopping rules Running Header: Approximate Farkas and Stopping Rules 1 1 Introduction This paper is concerned with what can be concluded about a linear programming problem and its dual when an in nite iterative method for its solution is terminated after a nite number of iterations. Our interest is mainly in recent interior-point methods, originating from the projective algorithm of Karmarkar [7]. Our concern is with practical methods that can be applied to problems with data given by real numbers; thus we do not consider methods whose initialization requires arti cial variables or arti cial constraints with \big M" coe cients related to the length of the input. If a variant of the simplex method (a nite algorithm) is run to completion, it provides relatively complete information about the linear programming problem and its dual, at least if we assume that exact arithmetic is used. If it does not generate optimal solutions to both problems, it shows why: it either demonstrates that the original problem is infeasible (and provides a short certi cate of its infeasibility via the well-known Farkas lemma) or proves similarly that the dual problem is infeasible (and thus the primal problem either infeasible or unbounded). For a discussion of this property of the simplex method and of the Farkas lemma, see for instance Schrijver [17]. When initial strictly feasible points are given (for the primal or for the primal and dual) then a (primal or primal-dual) interior-point algorithm will generate a sequence of strictly feasible points. If the method is terminated, feasible points are at hand, and usually some measure of their proximity to optimality is also available. Moreover, bounds are known for several methods on the number of iterations required to attain a given level of accuracy in the objective value. For a recent survey of many such methods, see Gonzaga [3]. This is a reasonably satisfactory situation. On the other hand, very often such initial feasible points are not provided or easily found, and then in practice a so-called infeasible-interior-point method is used. Such an algorithm generates a sequence of iterates that satisfy all nonnegativity constraints strictly, but may fail to satisfy equality constraints (general inequality constraints are converted to equations using nonnegative slack variables). When an algorithm of this type is terminated nitely, we usually have iterates that are in some sense approximately feasible and approximately optimal. But the original problem or its dual might be infeasible, and we would like the algorithm to provide some indication of this fact. Early infeasible-interior-point methods merely assumed that opti2 mal solutions existed. More recently, indications of the infeasibility of the problem or its dual, often in the form that, if optimal solutions did exist, they would have to be of very large norm, have been derived for certain algorithms. But the information provided is not totally satisfactory. There is little or no indication of which of the primal and dual problems is infeasible. This paper is an attempt to obtain more useful information from the output of such an iterative method. We are usually content with approximate answers when we look for optimal solutions. Thus the conclusion that we have nearly feasible solutions to the primal and dual problems with small total complementarity can be viewed as the availability of nearly optimal solutions to slightly perturbed problems; this is similar to backward error analysis in numerical analysis. Indeed, practical simplex implementations only provide this sort of information, since tolerances are used to compensate for rounding errors. We should ask for similar information when we stop with an indication of infeasibility; that is, we would like a proof that a slightly perturbed problem (or its dual) is infeasible. Since a proof of infeasibility for a system of linear inequalities is given by a solution to an alternative system as in the classical Farkas lemma, we are interested in approximate solutions to alternative systems and their interpretations for the original system. In Section 2, we state the linear programming problems we are concerned with and give two approximate Farkas lemmas that can be used to give indications of infeasibility or of feasibility. These results follow easily from more general results of Freund [2] and Renegar [16], but the form in which they are presented and the use we make of them seem to be new. In the next two sections we consider a generic infeasible-interior-point method and the homogeneous self-dual algorithm for linear programming problems with no known strictly feasible starting points. Algorithms of this type include those proposed by Kojima, Megiddo, and Mizuno [9], Lustig and Lustig, Marsten, and Shanno [10, 11], Mizuno [12], Potra [14, 15], Ye, Todd, and Mizuno [18], and Zhang [19]. We use the results of Section 2 to provide stopping rules which allow us to make de nitive conclusions about the problem and its dual, and we investigate the number of iterations required for some speci c algorithms before one of a set of stopping rules applies. If we seek either approximate optimality or an indication of infeasibility of the form that all feasible solutions of the problem or its dual (if any) must have large norm, then our results give better bounds for the homogeneous self3 dual algorithm than for other infeasible-interior-point methods; of course, this could be due to our analysis rather than di erences in the methods. The nal section includes some concluding remarks. 2 Approximate Farkas Lemmas We consider a linear programming problem given in the standard form minx cTx Ax = b x 0; (P ) with its dual written with explicit slacks as maxy;s bTy ATy + s = c s 0; (D) These problems are de ned by the matrix A 2 R I m n and the vectors b 2 R I m and c 2 R I n. We collect these into the data vector d := (A; b; c). We use arbitrary norms (both denoted by k k) on x 2 R I n and y 2 R I m, and the corresponding dual norms (denoted by k k ) on c and s in R I n and b and Ax in R I m. We also use the corresponding operator norm kAk := maxfkAxk : kxk 1g on A. While kAk may be hard to compute, we can easily evaluate the norms of rank-one perturbations. Indeed, if u 2 R I m and v 2 R I n, then kuvTk = kuk kvk . We de ne kdk := maxfkAk; kbk ; kck g. For de niteness, we sometimes restrict ourselves to the case where kxk := kxk1 (so that kck = kck1 and similarly for s) and kyk := kyk1 (so that kbk = kbk1). Then kAk = maxiPj jaijj. We call this choice for norms the standard case. We say d is primal feasible (infeasible) if the primal problem (P) de ned by the data in d is feasible (infeasible), and similarly for dual feasibility or infeasibility. In this section, we show that if (P) or (D) is \approximately infeasible" in the sense that we have an approximate solution to an alternative system as in the Farkas lemma, then any feasible solution must have large norm. Conversely, if (P) or (D) is \approximately feasible" in the sense that we 4 have an approximate solution to their equations and inequalities, then any vector proving their infeasibility must be large. As a corollary of these lemmas, we obtain versions of results in the very general theory of Renegar [16] on perturbation for linear programming. Our results can be proved from Renegar's general theory and from the dual gauge programs of Freund [2], but we include a proof of a part of each result directly to make the paper more self-contained. Both these results include the Farkas lemma. Below, we use the usual convention that the minimum over an empty set is +1; we also nd it useful to set 0 +1 and +1 0 conventionally to 1. Lemma 2.1 (Approximate Farkas Lemma I) Let x := minfkxk : Ax = b; x 0g; y := minfkyk : ATy + s = c; s 0g; and s := minfksk : ATy + s = c; s 0g: Let u := minfkuk : ATy u; bTy = 1g; v := minfkvk : Ax = v; cTx = 1; x 0g; and w := minfkwk : Ax = 0; cTx = 1; x wg: Then x u = y v = s w = 1. Proof. We just prove that x u = 1 in the case of the standard norms; the other proofs for this case are similar. First note that x is +1 i (P) is infeasible, which holds by the Farkas Lemma i u is 0. Similarly, x is 0 i b = 0, which holds i u is +1. Hence we may assume that both are positive and nite. Then x can be written as the optimal value of the linear programming problem minx; Ax = b x + e 0 x 0; and hence of its dual max~ y;~ u bT ~ y AT ~ y ~ u 0 eT ~ u = 1 ~ u 0: 5 Let ~ y and ~ u be an optimal solution to the latter problem. Since the optimal value is positive and nite, it is easy to see that y := ~ y =bT ~ y and u := ~ u =bT ~ y solve the problem de ning u, and hence that u = 1= x. 2 Remark. Our results follow from the application to lp-norm programming problems (which also holds for more general norms) of the gauge duality results of Freund; see p. 63 of [2]. In addition, Lemma 3.13 in Renegar [16] easily shows that x u 1. This is the hard part of the result; to show the reverse inequality (which can be viewed as a weak duality result), note that if x and (y; u) are feasible in the problems de ning x and u respectively, then kuk kxk uTx yTAx = bTy = 1: Similar short proofs hold for y v 1 and s w 1. There are also close connections between the lemma above and results bounding the distance from a given point to a polyhedron, rst due to Ho man [6]. A recent paper giving explicit bounds closely related to our results is G uler, Ho man, and Rothblum [4]; see also Dax [1]. This lemma has implications related to Renegar's results on perturbation of linear programming problems. Indeed, let us suppose rst that u is small. Then, if (y; u) solves the corresponding problem, we see that (A+ A)Ty 0; bTy = 1; where A := buT , with k Ak = kbk kuk = ukbk . Hence d is close to (indeed, within ukbk ukdk of) the primal infeasible data vector (A+ A; b; c). In the notation of [16], dist(d;Pri;) is at most ukbk . Using the lemma, we deduce that, if dist(d;Pri;) is positive, then there is a feasible solution to (P) with norm at most kbk =dist(d;Pri;); this is part (1) of Theorem 1.1 of [16]. Next, suppose v is small, and let (x; v) solve the corresponding problem. Then we nd that (A+ A)x = 0; cTx = 1; x 0; where now A := vcT , with k Ak = kvk kck = vkck . Hence d is now within vkck of a dual infeasible data vector, so dist(d;Dual;) vkck . The lemma then implies that, if dist(d;Dual;) is positive, then there is a 6 feasible solution to (D) with the norm of y at most kck =dist(d;Dual;); this is again an instance of the result quoted above. Finally, suppose w is small, with (x;w) a solution to the corresponding problem. Assume that wkck < 1=2, so that we may set x0 := (x+w)=(1 cTw) 0. We then nd that Ax0 = v; cTx0 = 1; x0 0; where now v := Aw=(1 cTw), with kvk kAkkwk=(1 kck kwk) 2 wkAk. We next proceed as in the previous paragraph to nd a nearby dual infeasible data vector; we obtain dist(d;Dual;) 2 wkAkkck . This implies that, as long as dist(d;Dual;) is positive, there is a feasible solution to (D) with the norm of s at most 2kAkkck =dist(d;Dual;). This last result is less satisfactory than the previous two because of the presence of both kAk and kck in the bound. This seems to be due to the way in which the vectors s and w enter into their respective problems. If we scale the problems initially so that kAk = 1 (if A is zero, the problems are trivial), kbk = 1 or b = 0, and kck = 1 or c = 0, then more expected \relative perturbation" results are obtained. Now we provide a complementary result. Lemma 2.2 (Approximate Farkas Lemma II) Let x := minfkxk : Ax = 0; cTx = 1; x 0g; y := minfkyk : ATy + s = 0; bTy = 1; s 0g; and s := minfksk : ATy + s = 0; bTy = 1; s 0g: Let u := minfkuk : ATy + s = c; s ug; v := minfkvk : Ax b = v; x 0g; and w := minfkwk : Ax = b; x wg: Then x u = y v = s w = 1. Proof. Again, we just prove that x u = 1 in the standard case; the other parts can be proved similarly for this case. The special case once again follows from the Farkas lemma: x is +1 i (D) is feasible, which holds i 7 u = 0. Also, x cannot be 0 and u cannot be +1. Thus we can assume that both are nite and positive. Then x is the optimal value of the linear programming problem minx; Ax = 0 cTx = 1 x + e 0 x 0; and hence of its dual max~ y;~ ;~ u ~ AT ~ y + c~ ~ u 0 eT ~ u = 1 ~ u 0: Let (~ y ; ~ ; ~ u ) be an optimal solution to the latter problem. Since the optimal value is positive and nite, it is easy to see that y := ~ y =( ~ ) and u := ~ u =( ~ ) solve the problem de ning u, and hence that u = 1= x. 2 This result shows that, if (D) is \almost feasible" ( u is small), then if it fails to be exactly feasible, any proof of its infeasibility must have large norm ( x is large); moreover, the dual-infeasible data vector d is close to the dualfeasible data vector (A; b; c + u). Similarly, if (P) is\almost feasible" ( v or w small), then, if it is not feasible, any certi cate of its infeasibility must be large in norm ( y or s large); moreover, d is close to the primal-feasible data vector (A; b + v; c) or (A; b+ Aw; c). This kind of argument has been used in the ellipsoid algorithm to relax the right-hand sides of a system of linear inequalities with integer data a small amount so that either both systems are feasible or both are infeasible; see Schrijver [17], p. 169. Our goal for an iterative algorithm to solve (P) and (D) is that it produce the following: either approximately feasible solutions x and (y; s) to (P) and (D) with xTs small; in this case we can conclude that we have almost optimal solutions to a linear programming pair that is close to the given one, and (using the second lemma above) that if (P) or (D) is infeasible, any proof of that fact via the Farkas lemma would have to have large norm; 8 or a vector x or a vector y that approximately demonstrates the infeasibility of (D) or (P) respectively; in this case we know that a problem near to (P) or (D) is infeasible, and (using the rst lemma) that if the corresponding original problem is feasible, any feasible solution must have large norm. 3 Infeasible-interior-point Algorithms Here we apply the results of the previous section to the output of the primaldual infeasible-interior-point method originally proposed by Lustig [10] and shown to be related to taking Newton steps to reach a point on the central path by Lustig, Marsten, and Shanno [11]. Kojima, Megiddo, and Mizuno [9] proved global convergence for a method of this type, and polynomial-time variants were analyzed by Zhang [19], Mizuno [12], and Potra [14]; see also [8, 15, 20]. We provide a number of stopping rules for a very general generic infeasibleinterior-point method, and discuss the conclusions that can be reached when these rules apply. Then, for more speci c methods, we give bounds on the number of iterations necessary until one of these rules holds. We suppose the algorithm generates a sequence fxkg of primal iterates and fyk; skg of dual iterates satisfying x0 = 0e; s0 = 0e; xk > 0; sk > 0; Axk b = k p(Ax0 b); ATyk + sk c = k d(ATy0 + s0 c); (xk)T sk p k p(x0)T s0; (xk)T sk d k d(x0)Ts0: (3.1) for some k p ; k d 2 [0; 1] and some positive constants 0; p, and d. For example, the algorithms in [12] and [19] satisfy these conditions with k p = k d for all k (because they take equal step sizes in the primal and dual problems) and p = d = 1. The algorithm of [14] furthermore has equality in the last two inequalities in (3.1) for all k. These conditions are not too restrictive. The requirement on the initial iterates is equivalent to starting at central points, where the products of corresponding components of x0 and s0 are all the same. In this case, by scaling the columns of A and hence the entries of the initial points, they can be brought into the form above. The second condition is natural for an 9 interior-point method, and the third follows if the iterates are generated by a line search using directions derived from some Newton system to achieve feasibility as well as centering and/or reduced complementarity. Finally, the last conditions seem to be necessary to derive convergence results; the total complementarity should not approach zero faster than primal or dual infeasibility. Below we state some stopping rules and their consequences. For simplicity, we omit the iteration superscript k in stating these rules. In what follows, all 's are (small) positive tolerances and all 's are (large) positive numbers. Stopping Rule 1a. Stop if kAx bk p; kATy + s ck d; and xT s o: In this case, clearly the current iterates are o-optimal in perturbed problems ( ~ P ) and ( ~ D), where b is perturbed by a vector of norm at most p and c by one of norm at most d. As mentioned at the end of the previous section, this implies that if the primal or dual problem is infeasible, any certi cate of its infeasibility using the Farkas lemma would have to have large norm. The second rule and the result and proof below are extensions of those of Mizuno [12]. They are also closely related to lemmas proved by Kojima [8]. Stopping Rule 2a. For some 0, stop if peTs+ deTx 0 +maxf 1 p ; 1 d g 20 xT s: Proposition 3.1 If stopping rule 2a applies, then there is no optimal solution pair x and (y ; s ) for (P) and (D) with kx k < and ks k < . Proof. Assume that such a pair exists. Using (3.1), we nd that A( px0+(1 p)x ) = Ax and AT ( dy0+(1 d)y )+( ds0+(1 d)s ) = ATy+s; so that [( px0 + (1 p)x ) x]T [( ds0 + (1 d)s ) s] = 0; or ( px0+(1 p)x )Ts+( ds0+(1 d)s )Tx = ( px0+(1 p)x )T ( ds0+(1 d)s )+xTs: 10 Hence 0( peTs+ deTx) ( px0 + (1 p)x )T s+ ( ds0 + (1 d)s )Tx = ( px0 + (1 p)x )T ( ds0 + (1 d)s ) + xTs < nmaxf p; dg 0 + xT s: But, using (3.1) again, p 1 p xTs=( 20n) and similarly d 1 d xTs=( 20n), so this last inequality gives a contradiction to the stopping rule. 2 This stopping rule and result do not lead to any indication of infeasibility, only that any optimal solutions will have to be large. We would like rules indicating likely infeasibility of either the primal or dual problems. For these, we let x̂ := AT (AAT ) 1b; ŷ := (AAT) 1Ac; and ĉ := c AT ŷ: Stopping Rule 3ap. Let ~ y := y [(1 d)ŷ + dy0]. Then stop if bT ~ y k(1 d)ĉ+ ds0k p: Stopping Rule 3ad. Let ~ x := x [(1 p)x̂+ px0]. Then stop if cT ~ x k(1 p)x̂+ px0k d: Proposition 3.2 If the condition in stopping rule 3ap holds, then any feasible solution to (P) has norm at least p; if that in stopping rule 3ad holds, then any feasible solution to (D) has ksk at least d. Proof. In the rst case, ATy + s c = (1 d)(AT ŷ + ĉ c) + d(ATy0 + s0 c) by (3.1), so that AT ~ y + s = (1 d)ĉ+ ds0: Thus ~ y=bT ~ y satis es the constraints of the problem de ning u in Lemma 2.1 with u := [(1 d)ĉ + ds0]=bT ~ y, which implies that u 1= p and hence x p. 11 In the second case, a similar argument using (3.1) shows that A~ x = 0, so that ~ x=( cT ~ x) is feasible in the constraints of the problem de ning w in Lemma 2.1 with w := [(1 p)x̂+ px0]=( cT ~ x). Hence w 1= d and thus s d. 2 For our next result, we assume that for each k, k p and k d are equal (usually corresponding to choosing equal step sizes in the primal and dual), and use k or just to denote their common value. We also suppose p and d are equal and write simply . We next show that, if is chosen su ciently large, then whenever is not too small and the \optimality unlikely" stopping rule 2a applies, then so does one of the \feasibility unlikely" rules 3ap and 3ad. Lemma 3.1 Suppose > 0 and 0 + 2 n maxfmax(kĉk = 0; kek ) p;max(kx̂k= 0; kek) dg: Then if stopping rule 2a applies, so does either 3ap or 3ad. Proof. We use repeatedly the fact that Ax = (1 )b + Ax0 and ATy + s = (1 )c+ (ATy0 + s0), from (3.1). Then xTs = (ATy + s)Tx (Ax)Ty = ((1 )c+ (ATy0 + s0)Tx (1 )b+ Ax0)Ty = (1 )(cTx bTy) + (y0)TAx+ (s0)Tx (x0)TATy = (1 )(cTx bTy) + (y0)T ((1 )b+ Ax0) (s0)Tx (x0)T ((1 )c + (ATy0 + s0)) = (1 )(cTx bTy) + 0(eTx+ eTs) 2 20n+ (1 )( bTy0 cTx0) = (1 )(cTx bTy) + 0(eTx+ eTs) 2 20n + (1 )( bTy0 + (1 )bT ŷ (1 )cT x̂ cTx0) = (1 )(cT ~ x bT ~ y) + 0(eTx+ eTs) 2 20n: Hence we have (1 )(cT ~ x bT ~ y) = xTs+ 2 20n 0(eTx+ eTs) (1 + 1 (1 + 1 0 ))xTs = 1( 0 1)xTs ( 0) 0n; 12 where we used the facts that 2 20n 20n = (x0)Ts0 1xTs and that stopping rule 2a applies. Since the right-hand side is nonpositive, we may remove the factor (1 ) on the left-hand side, and conclude that either bT ~ y ( 0) 0n=2 or cT ~ x ( 0) 0n=2: Using the formula for it is then easy to see that either stopping rule 3ap or 3ad applies. 2 Now we turn to the number of iterations required until one of the stopping criteria is activated for Potra's algorithm in [14]. While the steps of this algorithm are somewhat complicated (each step is composed of three substeps) it is simple in that the primal and dual infeasibilities and the total complementarity are all reduced at the same rate. Thus we write for the common value of p and d and choose p = d = 1. We also suppose for the rest of this section that the standard norms are used, so that kuk denotes the `1-norm and kuk denotes the `1-norm for u 2 R I n, while kvk denotes the `1-norm and kvk denotes the `1-norm for v 2 R I m. Let := 0 and := minf p kAx0 bk ; d kATy0 + s0 ck ; o (x0)T s0g: (3.2) Then the algorithm will terminate according to stopping rule 1a i . Moreover, if stopping rule 2a has not been activated, we can assume that eTs+ eTx 0 + 20 xTs; (3.3) and hence, using xT s = (x0)Ts0 = 20n, that 0(eTs+ eTx) (1 + ) 20n: We can now use this inequality in place of the inequality in Lemma 3.1 of (the revised version of) [14]. Suppose we choose our starting points by y0 = 0 or y0 = ŷ; x0 = s0 = 0e; 0 maxfkx̂k1; kck1; kĉk1g: (3.4) 13 Then we obtain, following the proof of Lemma 3.4 in [14], that the steps are bounded as stated there but with := := 2(1 ) 1=2(1 + )n: Then Lemma 3.5 of [14] remains true with these new values of and . We nd that (using the notation there) qj j and j j are O(n1=2 ) and hence that 4; 5; 6 and 8 are all O(n2( )2), and nally that 1= ̂ = O(n ). Thus is reduced at each iteration by a factor of the form 1 1=O(n ), so that O(n ln(1= )) iterations are su cient. To summarize, using our new estimates in the analysis of Potra we can establish the following: Theorem 3.1 Let Potra's algorithm be applied to (P) and (D) starting with the initial solutions given by (3.4), and let and be de ned by (3.2). Then either stopping rule 1a or stopping rule 2a will be activated within O(n ln(1= )) iterations. In particular, if we choose 0 = ( ), then = O(1) and O(n ln(1= )) iterations su ce. 2 It is important to realize that, if is large and we choose 0 = ( ), then is much smaller than p, d, and o. However, the added complexity is only a logarithmic term in . To be precise, if we assume that kAk, kbk1, and kck1 are O(1), then the nal bound can be replaced by O(n(maxfln(1= p); ln(1= d); ln(1= o)g+ ln )). Now suppose we do not want to terminate until we have either nearoptimal solutions or an indication that (P) or (D) is infeasible. We use Lemma 3.1. Suppose that the problems are scaled so that the right-hand side of (3.4) is O(1). We also suppose that p = d=n =: (this is reasonable since p is related to the `1-norm of x and d to the `1norm of s). Then the condition in Lemma 3.1 becomes 0 + (n ) 1n , where we have taken  = , or 1 + 1 0 : 14 Now we cannot achieve = O(1), since if we try to take 0 = ( ), or even larger, must be correspondingly smaller to obtain a comparable accuracy in feasibility and optimality. The best we can do is to take 0 = (1), and then = O( = ). Thus, using Lemma 3.1 and the theorem above, we get Theorem 3.2 Suppose Potra's algorithm is applied to (P) and (D) starting with the initial solutions given by (3.4), with 0 = (1), and let be de ned by (3.2). Then either stopping rule 1a or one of stopping rules 3ap or 3ad will be activated within O(n 1 ln(1= )) iterations. 2Note here the very unwelcome appearance of the terms  and 1, not logarithmically but as factors. To conclude this section, we examine Mizuno's algorithms in [12]. He also chooses equal step sizes so that p = d, but only requires the total complementarity to decrease at most as fast as the infeasibilities, rather than at the same rate. Thus (3.1) holds with p = d = 1. Suppose the starting points are again chosen by (3.4). Then kx1 u1k1 and kz1 w1k1, in the notation of [12], can be bounded by 2 0 (our 0 corresponds to 0 in [12], while our corresponds to there). Using this in the analysis above Lemma 3.3 in [12], and assuming (3.3) which implies eT s+ eTx ( 0) 1(1 + )xTs; we obtain a bound ofO(pn )pxTs on the norms of the scaled steps (kD 1 xk2 and kD zk2 in [12]). This yields complexities of O(n2( )2 ln(1= )) and O(n( )2 ln(1= )) steps for the rst and second algorithms in [12]: Theorem 3.3 Suppose Mizuno's rst (second) algorithm is applied to (P) and (D) starting with the initial solutions given by (3.4), and let and be de ned by (3.2). Then either stopping rule 1a or stopping rule 2a will be activated within O(n2( )2 ln(1= )) (O(n( )2 ln(1= )), respectively) iterations. In particular, if we choose 0 = ( ), then = O(1) and O(n2 ln(1= )) (O(n ln(1= )), respectively) iterations su ce. 2 If we want to stop with near-optimal solutions or an indication of infeasibility, we have some di culties using Lemma 3.1, since might be very small while stopping rule 1a still does not apply, because xTs is not su ciently small. Suppose we modify this rule, and consider 15 Stopping Rule 1a0. Stop if kAx bk p and kATy + s ck d: In this case, we have near-feasible solutions to (P) and (D). We can perturb b and c slightly and continue with a feasible-interior-point method from the current iterates until a near-optimal pair of solutions is obtained (compare with Section 5 of [9]). Now, if this rule has not been activated, we can assume that , and thus take  = in Lemma 3.1. Again we suppose that the problems are scaled so that the right-hand side of (3.4) is O(1), and that p = d=n =: : Then proceeding as above Theorem 3.2 we obtain Theorem 3.4 Suppose Mizuno's rst (second) algorithm is applied to (P) and (D) starting with the initial solutions given by (3.4), with 0 = (1), and let be de ned by (3.2). Then either stopping rule 1a0 or one of stopping rules 3ap or 3ad will be activated within O(n2 2 2 ln(1= )) (O(n 2 2 ln(1= )), respectively) iterations. 2 Once again we note the (even worse) polynomial, rather than logarithmic, appearance of  and 1 in the complexity bounds. 4 Homogeneous Self-dual Algorithms Here we apply the results of Section 2 to the output of an algorithm applied to the following homogeneous and self-dual (arti cial) linear program (HLP) relating (P) and (D) (this formulation was developed in [18]): (HLP ) min ((x0)T s0 + 1) s.t. Ax b + b = 0; (4:1) ATy +c c 0; (4:2) bTy cTx + z 0; (4:3) bTy + cTx z = (x0)T s0 1; (4:4) y free; x 0; 0; free; 16 where b = b Ax0; c = c ATy0 s0; z = cTx0 + 1 bTy0: (4.5) Here b, c, and z represent the \infeasibility" of the initial primal point, dual point, and primal-dual \gap", respectively. Let s 2 R I n and 2 R I denote the surplus in equations (4.2) and (4.3). By combining these equations ( (y0)T times the rst, (x0)T times the second, and ( 1) times the third and fourth), the last equality can be written as (s0)Tx+ (x0)T s+ + ((x0)Ts0 + 1) = (x0)Ts0 + 1; (4.6) which serves as a normalizing constraint for (HLP). Let Fh denote the set of feasible solutions to (4.1)-(4.4), along with the corresponding s and , such that all nonnegative variables are strictly positive. Note that, by construction, (y0; x0; 0 := 1; 0 := 1; s0; 0 := 1) 2 Fh. If (y; x; ; ; s; ) 2 Fh, then (y; x; ; ) is easily seen to be feasible in the dual of (HLP), with slacks s and (since (HLP) is self-dual), and so the duality gap for this pair of solutions equals the total complementarity, or (if we divide by two) ((x0)Ts0 + 1) = xTs+ =: (n+ 1) : (4.7) For simplicity, in what follows we assume x0 = s0 = e and y0 = 0, so that = from the equation above. We assume the algorithm generates a sequence f(yk; xk; k; k; sk; k)g of iterates satisfying (yk; xk; k; k; sk; k) 2 Fh and (4.8) k k (1 ) k = (1 ) k (4.9) for some xed 0 < < 1. The last requirement is a minimal centrality condition, and holds for instance for all path-following algorithms and some potential-reduction methods (see, e.g., [13]). Given such an iterate (y; x; ; ; s; ), the corresponding approximately feasible primal and dual solutions are x̂ := x= ; (ŷ; ŝ) := (y; s)= : We can now state some stopping criteria. 17 Stopping Rule 1b. Stop if kAx b k p ; kATy + s c k d ; and cTx bTy 0o : In this case, the corresponding primal and dual solutions as above are feasible in perturbed primal and dual problems, and the corresponding objective functions, using the original coe cients, are within 0o of each other. However, we might wish them to be 00 o-optimal in these perturbed problems, which suggests Stopping Rule 1b0. Stop if kAx b k p ; kATy + s c k d ; and xTs 00 o 2: It is not clear which of these two rules is more appropriate for a general termination criterion. It is easy to see that stopping rule 1b holds if pd := minf p k bk ; d k ck g and 0o max(0; z) : (4.10) Since xT s (n+ 1) = (n + 1) , stopping rule 1b0 applies if pd and  2 00 o n+ 1 : (4.11) The next rule applies when is small; we can then conclude that there are no small optimal solutions to (P) and (D): Stopping Rule 2b. Stop if 1 1 + : We have the following result. Proposition 4.1 If stopping rule 2b applies, then there is no optimal solution pair x and (y ; s ) for (P) and (D) with kx k1 + ks k1 < . 18 Proof. We follow the proof technique of G uler and Ye [5]. Assume that such a pair exists. Then y := y ; x := x ;  := ;  := 0; s := s ;  := 0; where = n + 1 eTx + eTs + 1 > 0; is an optimal solution to (HLP). Now we use (x x)T (s s) + (  )( ) = 0; which follows by subtracting the constraints of (HLP) for ( y; ; ) from those for (y; ; ) and using the skew-symmetry of the constraint matrix. This can be rewritten as xT s+ sT x+ +  = (n+ 1) : which implies (using (4.9)) that (n + 1)  1 n + 1  = 1 eTx + eTs + 1 > 1 1 + ; contradicting stopping rule 2b. 2 There is also a complementary result, although we do not use it in what follows. Proposition 4.2 Suppose there is a certi cate of primal infeasibility, i.e., (y ; s ) with ATy + s = 0; bTy = 1; s 0, or a certi cate of dual infeasibility, i.e., x with Ax = 0; cTx = 1; x 0, with kx k1 < or ks k1 < : Then, for all iterates, > 1 1 + : 19 Proof. We follow the argument above. Suppose there is such a \small" certi cate of primal infeasibility (the other case is similar). Then y := y ; x := 0;  := 0;  := 0; s := s ;  := ; where := n+ 1 eTs + 1 > 0; is optimal in (HLP). Now proceed as above with and interchanged to obtain the result. 2 The contrapositive of this result shows that, if we ever observe a small value of during the course of the algorithm, then there are no small certi cates of infeasibility for (P) or (D) and hence, using Lemma 2.2, there are almost feasible solutions to both problems. Suppose we use a feasible-interior-point algorithm to solve (HLP) that yields iterates satisfying (4.8-4.9) and guarantees in O(pn ln(1= )) iterations. Then we have Theorem 4.1 For such an algorithm, either stopping rule 1b or 2b will be activated withinO pn ln[maxf1 + pd ; max(0; z)(1 + ) 0o g]! iterations. Also, either stopping rule 1b0 or 2b will be activated within O pn ln[maxf1 + pd ; n(1 + )2 00 o g]! iterations. Proof. Since (1=((1 + )) while stopping rule 2b fails to hold, the result follows from (4.10) and (4.11). 2 Note that the dependence on is only logarithmic in these bounds, as in the previous section. Once again, stopping rule 2b only gives an indication that any optimal solutions of (P) and (D) must be large, rather than a suggestion of infeasibility. As in the previous section, we can add rules that indicate that any feasible solution must be large. 20 Stopping Rule 3bp. Stop if bTy ( kck + k ck ) p: Stopping Rule 3bd. Stop if cTx ( kbk + k bk ) d: Proposition 4.3 If the condition in stopping rule 3bp holds, then any feasible solution to (P) has norm at least p; if that in stopping rule 3bd holds, then any feasible solution to (D) has kyk at least d. Proof. The proof is similar to that of Proposition 3.2. For the rst part, set y := y=bTy. Since ATy + s = c c, we see that y is feasible in the problem de ning u in Lemma 2.1 with u := ( c c)=bTy, which implies u 1= p and hence x p. For the second part, note that cTx is negative and so x := x=( cTx) is feasible in the problem de ning v in Lemma 2.1 with v = ( b b)=( cTx), which implies v 1= d and hence y d. 2 We want to bound the number of iterations until we get near-optimal solutions or an indication of likely infeasibility of the primal or dual problem. Let := maxf pkck ; pk ck ; dkbk ; dk bk ; zg (4.12) and := minf2 3 ; pd; 0o max(0; z)g: (4.13) Theorem 4.2 If (1 ) 2 4 ; (4.14) then one of stopping rules 1b, 3bp, and 3bd holds. Proof. Suppose stopping rule 1b does not apply. Then we know that = > . Thus, using (4.9), we have (1 )( = ) > (1 ) . Now use 21 the bound on to get> 4[1 + 2(1 + 1)]z + 2(1 + )z + 1[( kck + kck )p + ( kbk + k bk )d]:But note that bTy cTx = z , so the inequality above shows that one ofstopping rules 3bp and 3bd holds. 2If we use a feasible-interior-point algorithm to solve (HLP) that yieldsiterates satisfying (4.8-4.9) and guaranteesinO(pn ln(1= )) iterations,we obtain from the theorem above that one of stopping rules 1b, 3bp, and3bd will be activated within O(pn ln(= )) iterations. Note that here, incontrast to the situation for the infeasible-interior-point algorithms analyzedin Section 3 (see Theorems 3.2 and 3.4), the dependence on  and 1 islogarithmic.Unfortunately, we see no way to bound the number of iterations until oneof stopping rules 1b0, 3bp, and 3bd holds, if we apply the algorithm directly.The reason is that, according to (4.11), stopping rule 1b0 might not hold eventhough = is very small, as long as = 2 is still too large. However, thisseeming drawback is not critical. As we mentioned above, it is not clear whichof rules 1b and 1b0 is more appropriate in a particular situation. Secondly,even if we want to satisfy the condition in rule 1b0, we can terminate thealgorithm applied to (HSD) when satis es (4.14) (in a number of iterationsas given above), setting0o very large. Then, if one of rules 3bp and 3bd applies,we stop. Otherwise, =  and rule 1b applies, so we have nearly feasibleprimal and dual solutions. We can then replace b and c by their perturbedvalues and continue with a standard feasible-interior-point method, similarto what we did to derive Theorem 3.4 for Mizuno's algorithm. Since we havexT s2 (n+ 1)2(n+ 1)(n+ 1)24(n+ 1)1;we only need another O(pn ln(n= 00o)) iterations to satisfy the criterion instopping rule 1b0. Thus, one of stopping rules 1b0, 3bp, and 3bd will be acti-22 vated within O(pn ln(=̂)) iterations in total, wherê :=minf23 ; pd; 00on g:Again, the dependence on  and ̂ 1 is logarithmic.5 Concluding RemarksIn Section 2, we gave two general lemmas that make precise how certain near-feasible solutions to systems of inequalities indicate the likely infeasibility ofalternative systems, extending the Farkas lemma. We used these lemmas tosuggest what an iterative method for linear programming problems shouldproduce as its output.In the last two sections, we described a number of stopping rules forinterior-point algorithms. We showed that certain combinations of thesestopping rules can lead to one of two conclusions. The rst is that either anear-optimal pair of solutions to a linear programming problem and its dual isat hand, or alternatively that all optimal solutions have large norm. The sec-ond replaces the second alternative with a certi cate that all feasible solutionsto the primal, or all feasible solutions to the dual, have large norm. Previousstudies on interior-point methods have generally only been able to give con-clusions of the rst type. We have given bounds on the number of iterationsrequired before one of these conclusions can be made. For infeasible-interior-point algorithms, this bound is polynomial, rather than logarithmic, in thesize of the large norm in the case that achieving near-optimality or indicatinginfeasibility is the goal. In contrast, applying a feasible-interior-point algo-rithm to the homogeneous self-dual formulation provides such conclusions ina number of iterations that is logarithmic in this size.References[1] A. Dax. On minimum norm solutions. Journal of Optimization Theoryand Applications, 76:183{193, 1993.23 [2] R. M. Freund. Dual gauge programs, with applications to quadraticprogramming and the minimum-norm problem. Mathematical Program-ming, 38:47{67, 1987.[3] C. C. Gonzaga. Path following methods for linear programming. SIAMReview, 34(2):167{227, 1992.[4] O. Guler, A. J. Ho man, and U. G. Rothblum. Approximations tosolutions to systems of linear inequalities. To appear in SIAM Journalon Matrix Analysis and Applications, 1995.[5] O. Guler and Y. Ye, Convergence behavior of interior-point algorithms.Mathematical Programming, 60:215{228, 1993.[6] A. J. Ho man. On approximate solutions of systems of linear inequali-ties. Journal of Research of the National Bureau of Standards, 49:263{265, 1952.[7] N. K. Karmarkar. A new polynomial{time algorithm for linear program-ming. Combinatorica, 4:373{395, 1984.[8] M. Kojima. Basic lemmas in polynomial-time infeasible-interior{pointalgorithms for linear programs. Research Reports on InformationSciences, Ser. B : Operations Research B{268, Dept. of InformationSciences, Tokyo Institute of Technology, Oh{Okayama, Meguro{ku,Tokyo 152, Japan, June 1993.[9] M. Kojima, N. Megiddo, and S. Mizuno. A primal{dual infeasible-interior-point algorithm for linear programming.Mathematical Program-ming, 61:263{280, 1993.[10] I. J. Lustig. Feasibility issues in a primal{dual interior point method forlinear programming. Mathematical Programming, 49:145{162, 1990/91.[11] I. J. Lustig, R. E. Marsten, and D. F. Shanno. Computational experiencewith a primal{dual interior point method for linear programming. LinearAlgebra and Its Applications, 152:191{222, 1991.24 [12] S. Mizuno. Polynomiality of infeasible interior{point algorithms for lin-ear programming. Technical Report, The Institute of Statistical Math-ematics, 4{6{7 Minami{Azabu, Minato{Ku, Tokyo 106, Japan, May1992, revised January 1993.[13] S. Mizuno, M. J. Todd, and Y. Ye. On adaptive step primal{dualinterior{point algorithms for linear programming. Mathematics of Op-erations Research, 18:964{981, 1993.[14] F. A. Potra. An infeasible interior{point predictor{corrector algorithmfor linear programming. Reports on Computational Mathematics 26,Dept. of Mathematics, The University of Iowa, Iowa City, IA 52242,USA, June 1992, revised August 1994, to appear in SIAM Journal onOptimization.[15] F. A. Potra. A quadratically convergent predictor-corrector method forsolving linear programs from infeasible starting points. Reports on Com-putational Mathematics 28 and 34, Dept. of Mathematics, The Univer-sity of Iowa, Iowa City, IA 52242, USA, July and October 1992, revisedDecember 1993, to appear in Mathematical Programming.[16] J. Renegar. Some perturbation theory for linear programming. Mathe-matical Programming, 65:73{82, 1994.[17] A. Schrijver. Theory of Linear and Integer Programming, J. Wiley,Chichester, 1986.[18] Y. Ye, M. J. Todd, and S. Mizuno, An O(pnL)-iteration homogeneousand self-dual linear programming algorithm. Mathematics of OperationsResearch, 19:52{67, 1994.[19] Y. Zhang. On the convergence of a class of infeasible interior{point meth-ods for the horizontal linear complementarity problem. SIAM Journalon Optimization, 4:208{227, 1994.[20] Y. Zhang and D. Zhang. On polynomiality of the Mehrotra-typepredictor-corrector interior-point algorithms. Research Report 93{12,Dept. of Mathematics and Statistics, University of Maryland BaltimoreCounty, Baltimore, MD 21228{5398, USA, revised June 1994.25